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(54) Image data processing apparatus having functions of dividing the image data, and method 
of processing image data 



(57) The image data processing apparatus com- 
prises a camera (S2) for converting an image of an area 
to be processed into image data, a function (S3) of divid- 
ing the image data into divisional image data items cor- 
responding to divisional areas obtained by dividing the 
divisional areas, and a function (S4) of performing pre- 
determined processing for each of the divisional image 
data items, thereby to detect an abnormal state of the 
area to be processed. 
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Description 

The present invention relates to an abnormal state 
detect method and an abnormal state detect apparatus, 
for monitoring an image within a monitor area with use 5 
of an image pick-up means such as an I TV camera, to 
detect an abnormal state in an image, and an abnormal 
state detect apparatus. 

For example, in case where an image pick-up means 
is used to monitor an image within a monitor area, a con- 10 
secutive difference of images inputted to be consecutive 
on the time base or a difference between an image and 
a background image. Then, the result is subjected to 
binary processing, to extract candidates of change areas 
which are considered to be caused by a movement of an is 
object or the like. 

In this case, areas where an object does not spatially 
exist are previously checked from placement conditions 
of an ITV camera, and data concerning such an object 
which has been previously recognized as will sure to 20 
enter into the monitor area is previously stored as a tem- 
plate. An change area which is too small among the 
extracted change areas is considered as noise and is 
removed, while the other remaining candidate areas are 
subjected to comparison with a template. If any of the 25 
areas is similar to the template, it is determined that the 
change of the area has already been recognized. If there 
is not an area similar to the template, the change of the 
area is a change which has not been previously recog- 
nized, and therefore, determined as an abnormal state. 30 

Further, in several cases, this apparatus is applied 
to a paper material dealing apparatus or the like which 
classifies stocks, mails, and the likes by conveying them 
through convey paths. This card material dealing appa- 
ratus conveys postcards through convey paths, thereby 35 
classifying the paper materials, depending on their clas- 
sifications, and paper materials to be dealt with are 
moved through substantially fixed lines previously pre- 
determined. These paper materials sometimes derive or 
fall from the convey paths, due to accidents; due to 40 
abnormal states, e.g., bending or break-down of the 
paper materials or due to environmental changes. How- 
ever, if an abnormal state detect apparatus is placed on 
a convey path, such abnormal states can be detected. 

A conventional abnormal state detect method is, 45 
however, easily influenced by noise caused by vibrations 
of an ITV camera or the like, and influences from such 
noise appear when a differential binary image between 
images is obtained. Therefore, there is a problem that 
any noise removing processing must be provided. so 

In addition, it is necessary to previously provide 
information concerning the position where an ITV cam- 
era is placed. Further, when extraction is performed from 
candidate areas obtained as a result of differential bina- 
rize processing, comparison with a number of templates 55 
corresponding to objects which can occur in an image 
must be carried out. Therefore, there is a problem that 
the number of calculations tends to increase. 
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Further, since many kinds of templates are required 
with respect to object which can occur in order to make 
determination based on the shape of an change area 
when whether a change occurring is caused by a factor 
of an abnormal state or is included in a regular state is 
determined, there is a problem that it is not easy to dis- 
tinguish a regular change from an abnormal change 
state. 

In addition, when this apparatus is applied to a paper 
material dealing apparatus as described above, an oper- 
ator must periodically carry out a complicated inspection 
as to whether or not an ITV camera is placed a monitor 
or area, whether or not there is a defective in the abnor- 
mal state detect function. 

The present invention has an object or providing an 
image data processing method and an image data 
processing apparatus capable of detecting changes and 
abnormal states in an image at a high speed, without 
comparison with templates. 

The present invention has another object of provid- 
ing an image data processing method and an image data 
processing apparatus capable of automatically setting a 
position of a detection area and diagnosing a detect func- 
tion, without an operator. 

The present invention provides an image data 
processing apparatus comprising: means for continu- 
ously receiving a image of a first area to be processed, 
and for converting the image into a first image data item; 
means for dividing the first image data into a plurality of 
second image data items corresponding to a plurality of 
second areas which are smaller than the first area; and 
means for performing predetermined processing for 
each of the second image data items so as to determine 
an abnormal state in the first area. 

The present invention further provides an image 
data processing method according to the first embodi- 
ment comprising: an inverting step of continuously 
receiving an image of a first area to be processed, and 
of converting the image into a first image data item; a 
dividing step of dividing the first image data into a plural- 
ity of second image data items corresponding to a plu- 
rality of second areas which are smaller than the first 
area; and a detecting step of performing predetermined 
processing for each of the second image data items, 
thereby to determine an abnormal state in the first area. 

Further, the present invention provides an image 
data processing apparatus comprising: means for con- 
tinuously receiving an image of a first area to be proc- 
essed, and for converting the image into a first image 
data item; means for dividing the first image data into a 
plurality of second image data items corresponding to a 
plurality of second areas each of which is smaller than 
the first area; means for generating a recognizable mark 
in the first area at a predetermined timing; and means 
for detecting the mark generated by the means for gen- 
erating a mark, thereby to determine that an inverting 
function of the image data processing apparatus regu- 
larly operates. 
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As described above, according to the image 
processing apparatus and method of the present inven- 
tion, an image to be processed is divided into a plurality 
of images, and predetermined processing is performed 
on these divisional images, thereby to determine 5 
whether an abnormal state (or a change) is included in 
the image. Said predetermined processing means, for 
example, processing of determining that an abnormal 
state occurs in an image if the average density of an 
image or a time-based change in the characteristic w 
amount of a dispersion value is a predetermined value 
or more. Due to this processing, an abnormal state can 
be detected by merely detecting changes in the charac- 
teristic amounts, so that processing for comparing an 
image with image templates as is used in a conventional 75 
apparatus is not required any more. As a result, the 
processing time can be greatly reduced, in addition, 
vibration of a camera is absorbed, and therefore, coun- 
termeasures for noise is not necessary any more. 

In addition, according to the invention as described 20 
above, a recognizable mark is generated (e.g., by an 
LED) in the area of an image to be processed, and this 
mark is recognized and determined by an image pick-up 
system. As a rest of this, an operator need not carry out 
complicated preparation for detection as to whether or 25 
not an abnormal state exists in the functions of the image 
pick-up system, unlike in a conventional apparatus. 

Further, according to the present invention, if pro- 
grams are arranged such that self-diagnosis processing 
for the image pick-up system is automatically carried out, 30 
a defect can be detected early unless an operator posi- 
tively carries out diagnosis. 

This invention can be more fully understood from the 
following detailed description when taken in conjunction 
with the accompanying drawings, in which: 35 

FIG. 1 is a block diagram schematically showing the 
structure of an abnormal state detect apparatus accord- 
ing to an embodiment of the present invention. 

FIG. 2 is a view showing an example of a monitor 
environment and placement of an ITV camera. 40 

FIG. 3 is a view which explains processing for divid- 
ing an input image into a plurality of areas. 

FIG. 4 is a view which explains how parameters of 
areas are provided. 

FIG. 5 is a view showing an area dividing method. 45 

FIG. 6 is a view showing changes in characteristic 
amounts of an input image and areas. 

FIG. 7 is a flow-chart which explains the entire 
processing. 

FIG. 8 is a flow-chart which explains processing in so 
an area level. 

FIG. 9 is a flow-chart which explains the processing 
in the entire areas. 

FIG. 10 shows examples of time-based changes of 
characteristic amounts. 55 

FIG. 1 1 A and 1 1 B explain an example of the manner 
in which characteristic parameters are decided. 

FIG. 12 is a flow-chart which explains the entire 
processing of an abnormal state detect apparatus 



according to a second embodiment of the present inven- 
tion. 

FIG. 13 shows an example of a time-based change 
of a characteristic amount such as noise. 

FIG. 14 explains an example of area division. 

FIG. 15 shows an example of a time-based change 
of a characteristic amount which does not include sub- 
stantial changes. 

FIG. 16 shows an example of area fusion. 

FIG. 17 is a flow-chart which explains division 
processing of areas. 

FIG. 18 is a flow-chart which explains fusion 
processing of areas. 

FIG. 19 is a flow-chart which explains fusion 
processing of areas. 

FIG. 20 schematically shows a structure of a paper 
material dealing apparatus according to a third embodi- 
ment of the present invention. 

FIG. 21 is a block diagram of an example of a struc- 
ture mainly showing a processing device according to the 
third embodiment. 

FIG. 22 explains a specific example of a mark gen- 
erating section. 

FIG. 23 explains another specific example of a mark 
generating section. 

FIG. 24 is a flow-chart which explains self-diagnosis 
processing of an image pick-up system. 

FIG. 25 is a block diagram of an example of a struc- 
ture mainly showing a processing device according to a 
fourth embodiment of the present invention. 

FIG. 26 explains an specific example of a false 
abnormal state generating section. 

FIG. 27 is a flow-chart which explains self-diagnosis 
processing of a detect function. 

FIG. 28 shows an example of placement of a mark 
within a set monitor area according to a fifth embodiment 
of the present invention. 

FIG. 29Ato FIG. 29C explain how a mark is viewed 
in a picked up image. 

FIG. 30A is a flow-chart explaining an operation for 
setting a monitor area. 

FIG. 30B is a flow-chart explaining an operation for 
setting a monitor area. 

In the following, embodiments of the present inven- 
tion will be explained with reference to the Drawings. 

A first embodiment of the present invention will firstly 
be explained below. 

FIG. 1 schematically shows the structure of an 
abnormal state detect apparatus according to the first 
embodiment. Specifically, an ITV camera 10 as an image 
pick-up means is placed, for example, as shown in FIG. 
2, and images within a monitor area E as shown in the 
figure are continuously picked up, and converted into 
electric signals. Image signals (analog signals) outputted 
from the ITV camera 10 are inputted into an A/D convert- 
ing section 21 , and are digitalized by sampling pulses of 
a predetermined sampling rate outputted from a sam- 
pling pulse generating section 22. The signals thus digi- 
talized are sequentially stored as image data into an 
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image memory 23. These input images can be recog- 
nized by displaying them on a display device 13, and 
also, can be inquired layer if these input images are 
recorded and stored by a recording device 24 such as a 
video tape recorder or the like. 5 

A processing section 25 is constituted mainly by a 
CPU, and performs processing as will be described later 
on a plurality of images (e.g., two images) taken in on 
the time-base by an image memory 23, thereby to 
achieve processing for extracting an abnormal state from w 
the images. If it is determined that an abnormal state 
occurs, the processing section 25 makes an alarm 
device 26 to operate to notify an operator or the like of 
the abnormal state, while the processing section 25 
makes the recording device 24 record images preceding is 
and following the image determined as including an 
abnormal state, and makes those images displayed on 
the display device 13, so that an operator can confirm 
the abnormal state. 

The processing section 25 divides an input image 20 
(of MxN pixels) as shown in (a) on FIG. 3 taken in by the 
image memory 23 into a plurality of areas, such that each 
of the area has an equal defined by uniformly dividing an 
input image in the longitudinal and lateral directions (e.g., 
by m and n, respectively). The areas thus divided respec- 25 
tively have their numbers with respect to the total number 
of the entire areas, and their coordinate values of left 
upper and right lower corners, as parameters (see FIG. 
4)- 

With respect to division of areas, there is a method 30 
in which areas do not overlap each other as shown in (a) 
on FIG. 5 and another method in which areas partially 
overlap each other. Although each of divided areas have 
been explained as having a size equal to each other, the 
size of such an area which should particularly be moni- 35 
tored finely may further be divided into a number of small 
areas, and thus, it is possible to change the size of the 
divided areas if necessary. In this case, areas must be 
assigned to the entire image. 

To actually process an image, an input image at 40 
each time is divided into a plurality of areas as has been 
explained above, and each of the areas is subjected to 
the processing as will be explained below. Although divi- 
sion of areas has been explained to be performed such 
that each of divided areas has a rectangular shape, the 45 
shape of an area is not limited to a rectangle in any 
cases. 

In the following, detailed explanation will be made to 
time-based changes in characteristic amount in each 
divided areas and processing for extracting an abnormal so 
state. 

FIG. 6 shows an example in which an input image 
is divided into a plurality of areas, and thereafter, an 
abnormal state is extracted on the basis of time-based 
changes in the characteristic amount in the divided 55 
areas. This figure shows an area which includes a 
change in accordance with changes in characteristic 
amounts within the divided areas. The flow of this 
processing will be explained with reference to a flow- 



chart of the entire processing shown in FIG. 7. In FIG. 6, 
(a) to (d) show input images, while (e) to (g) show 
changes in characteristic amounts of areas correspond- 
ing to the input images. 

At first, with respect to each of divided areas, time- 
based changes in characteristic amount is set as a 
parameter in initial setting (S1). Next, an input image 
converted into digital image data at each of time points 
is taken in (S2) f and the input image thus taken in is 
divided into a plurality of areas (S3). With respect to each 
of the area, any characteristic amount included in each 
image is calculated (S4). In each of the areas, changes 
in characteristic amount up to the time point are com- 
pared with a parameter supplied to the area, presence 
or absence of an abnormal state is checked for every 
area (S4). In the next, results of processing of respective 
areas are integrated, and presence or absence of an 
abnormal state is checked with respect to the entire 
areas. If it is determined that an abnormal state occurs, 
notification is performed (S5). 

Here, an average density value of an image, a dis- 
persion value, and a confusion (complication) using a 
result of differential binarization may be used as the char- 
acteristic amount included in each divided area. In case 
of a color image, each color phase, a chroma, a bright- 
ness may be used. 

If an average density value or a dispersion value are 
used as the characteristic amount, an average value 
within an area is once obtained and a dispersion value 
is calculated again, according to a normal dispersion 
value calculation method. However, if an taking-in inter- 
val between images is short and the processing speed 
is high, it is considered that a change in an image pick- 
up view field between consecutive image is not relatively 
large, so that it is possible to obtain a dispersion value 
with use of an average value at a preceding time point, 
i.e., to use a so-called false dispersion value. 

In the next, the processing of extracting an abnormal 
state at respective area levels in the step S4 in the flow- 
chart of FIG. 7 will be specifically explained with refer- 
ence to the flow-chart shown in FIG. 8. 

A characteristic amount d(t) is extracted sequentially 
for every of the divided areas (S1 1 ), and a difference d(t- 
1) between the extracted characteristic amount at a pre- 
ceding time point is obtained. If it is determined that a 
change exists by threshold value processing (S12), the 
characteristic amount is compared with a parameter 
(S13). If the characteristic amount exceeds a predeter- 
mined threshold value as a result of comparison with an 
allowable range of the parameter (Si 4), it is determined 
that the change in characteristic amount is caused by an 
abnormal state, i.e., an abnormal state exists in the cor- 
responding area (S15). This processing is performed on 
all of the areas. 

Note that the threshold value used for extracting a 
change as described above and a threshold value of an 
allowable range of a parameter may be experimentally 
set with use of a result obtained by previously checking 
changes in characteristic amount with use of a test 
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image, or may appropriately be obtained such that 
changes in characteristic amount can be extracted in the 
procedures of processing steps. 

Examples of such threshold values will be explained 
below. s 

In FIG. 10, (a) is taken as an example of a time- 
based change in characteristic amount within each area. 
In this case, six parameters are considered as shown in 
FIG. 1 1 A , e.g., a value (B) where the amount is stable, 
a height (H) when a change occurs, a continuation period w 
(L) for which a change is maintained, and errors p, r), and 
X. 

These parameters may be determined by investiga- 
tion a result obtained by analyzing image data within a 
monitor area E, or may be determined from a result is 
obtained by arranging a processing system so as to have 
a learning function and making the processing system 
to perform learning such that a time-based change in 
characteristic amount can be appropriately extracted as 
described above. 20 

Six examples shown in FIG. 10, (b) to (g) are sup- 
posed as time-based changes in characteristic amount 
in this area. 

On (b) in FIG. 10, the change is remarkable at a 
starting period, and the amount soon returns to an orig- 25 
inal stable state. Therefore, this change is simply consid- 
ered as noise. 

On (c) in FIG. 10, the entire shape and the continu- 
ation period are substantially equal to those of the above 
case, but the height of the peak exceeds the allowable 30 
range. Therefore, this change considered as an abnor- 
mal state. 

On (d) in in FIG. 10, the change is remarkable at a 
starting period, and the amount returns to an original 
state after a time period substantially equal to a contin- 35 
uation period. In this state, the change is similar to a 
parameter previously prepared, and is therefore consid- 
ered as a regular change. 

On (e) in FIG. 10, the change is remarkable at a 
starting period, and the amount once decreases before 40 
the continuation period ends. Then, the amount returns 
to the height of the peak, and finally, returns to an original 
state after a time period substantially equal to the con- 
tinuation period. This is a change similar to a parameter 
previously included, and is therefore considered as a 45 
regular change. 

On (f) in FIG. 1 0, the change is remarkable at a start- 
ing period, and the continuation period starts. However, 
the amount changes over the height of the peak. The 
change thus differs from a parameter previously so 
included, and is therefore determined that any abnormal 
state exist. 

On (g) in FIG. 10, the change is remarkable at a 
starting period, and the continuation period starts. How- 
ever, the amount does not return to an original state after 55 
the continuation period. Therefore, there is a possibility 
that something is staying in the corresponding area, so 
that it is determined that an abnormal state exists. 
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As explained above, if a change which is determined 
as an abnormal state exists in changes in an area, it is 
determined that an abnormal stat is present in the cor- 
responding area. 

Note that the six parameters as described above 
may be supplied for all of the areas, supposing that any 
change occur in all of the areas. However, parameters 
other than B need not be supplied for those areas which 
have previously been recognized as including no 
changes, since parameters other than B are not neces- 
sary for such areas. Further, in case where an object 
which cyclically moves (e.g, an object conveyed on a belt 
conveyer of a conveyer system) is monitored, another 
parameter S(±a) may be supplied as an occurrence 
interval between changes. 

In the above explanation, a rectangular wave which 
changes in the positive direction is used as an example 
of a time-based change in characteristic amount. It is 
needless to say that corresponding processing is per- 
formed in other cases. 

Next, processing of extracting an abnormal state at 
the entire area levels in the step 5 in the flow-chart shown 
in FIG. 7 will be specifically explained with reference to 
the flow-chart shown in FIG. 9. 

After the processing for each area is completed, to 
whether or not the processing for the entire areas is nec- 
essary is checked (S21). If the processing for the entire 
areas is necessary, results of areas in the periphery of 
an area indicating an abnormal state among the entire 
areas are investigated (S22). If there is another area 
including an abnormal state (S23) in the periphery of the 
area indicating an abnormal state, whether or not there 
is further another area indicating an abnormal state in 
the periphery of said another area is checked (S24), in 
the same way. If there is an area indicating an abnormal 
state, these areas are combined with each other (S25). 
In this state, if the areas indicating an abnormal state 
exist isolated from each other, the size of an area chang- 
ing is too small, the abnormal state is considered as 
noise (S26). 

If the number of elements when areas indicating an 
abnormal state are finally combined each other exceeds 
a predetermined threshold value (S28) after presence of 
an abnormal state is thus checked with respect to the 
entire areas (S27), the group of those areas is deter- 
mined as an abnormal area (S29), and it is determined 
that an abnormal state exists in the image. A notification 
is then performed by stopping processing. Even if the 
predetermined threshold value is not exceeded, any 
measure is taken for attracting an attention. 

The following methods are considered as a method 
of determination as to whether or not the processing 
should be performed with respect to the entire areas in 
the step S21. Specifically, as described above, occur- 
rence of an abnormal state is checked for each area, and 
thereafter, searching for an abnormal state is performed 
on the entire areas where one of the following conditions 
is satisfied. 
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(1) As a result of performing processing of each 
area, one or more areas indicating an abnormal 
state exist. 

(2) The ratio of the number of areas determined as 
including an abnormal state to the number of the s 
entire areas exceeds a predetermined value. 

Note that, if the processing speed is high, each of 
changes in the image pick-up view field is not considered 
as large, and it is therefore possible to obtain a dispersion 10 
value with use of an average value at a preceding time 
point, i.e., a so-called false dispersion value, without 
using a normal calculation method of obtaining once an 
average value within an area and then calculating a dis- 
persion value again, when a dispersion value is calcu- is 
lated. 

In the next, a second embodiment will be explained. 

The second embodiment is different from the first 
embodiment in that the processing (S5) for the entire 
areas, which is performed in the first embodiment, is not 20 
carried out, and instead, area division and fusion 
processing (S6) are performed, as shown in the flow- 
chart of the entire processing in FIG. 12. The area divi- 
sion and fusion processing will be specifically explained 
bellow. 25 

At first, processing of area division will be explained. 
As shown in FIG. 13, a time-based change in character- 
istic amount within a divided area shows a noise-like 
state, it is determined that noise always exists in the cor- 
responding area. To prevent such determination, each 30 
of the divided areas is further divided into small areas, 
so that time-base changes in characteristic amount can 
be monitored. 

Specifically, with respect to areas in an image, if a 
time-based change in characteristic amount within an 35 
area is similar to noise as shown in FIG. 13, if the fre- 
quency of occurrences of changes is high, and if this 
state continues for a predetermined time period or more, 
the area is further divided. 

With respect to this processing of area division, 40 
explanation will be made with reference to the flow-chart 
shown in FIG. 17. After extraction of characteristic 
amount d(t) is completed with respect to each of areas 
divided in the step S4, a difference between the amount 
and a characteristic amount d(t-1) at a preceding time 45 
point is obtained, and it is determined that a change 
occurs if the difference data is larger then a threshold 
value Ts (S31). The number of times for which the 
change occurs is calculated. If this change continuously 
exists and the number of times for which the change so 
occurs exceeds a predetermined threshold value Tcs 
(S33), area division is performed (S34). 

As a result of this, since parameters are not previ- 
ously set with respect to those areas which are newly 
added by the area division, parameters of the areas ss 
before this area division are assigned to those areas 
(S35). In the next, the number of times (Cs) for which a 
change occurs is reset (S36), and the flow goes to the. 
next processing. 



In the area division in the step S34, for example, divi- 
sion as shown in FIG. 14 is performed. In this step, an 
division is performed such that an area is divided into 
four equal small areas. A number of a new area and coor- 
dinate values of the left upper and right lower corners 
are assigned, for example, as parameters to each of the 
divided areas. In addition, depending on the division 
method of the initial state, areas may be divided so as to 
have an overlapped portion or not to have an overlapped 
portion (see FIG. 5). 

In the next fusion processing of areas will be 
explained. In case where time-based changes in char- 
acteristic amount in an area are not substantially 
observed as shown in FIG. 15, with respect to areas in 
an image, such changes occur at a low occurrence fre- 
quency, and this state continues for a predetermined 
period or more, fusion with an area which is in a similar 
state in the periphery of the area is performed. 

This processing of area fusion will be explained with 
reference to the flow-chart shown in FIG. 18. After com- 
pletion of division processing of the above described 
areas, a state counter for counting absence of changes 
counts "+1" (S41). Then, a difference between the char- 
acteristic amounts d(t) and d(1 -t) extracted as described 
above is obtained. If the difference data is larger than a 
threshold value Tm (S42), it is determined that a change 
occurs, and the state counter is reset (S43). The flow 
goes to processing of a next image. 

When a state where no changes exists continues 
and the count value of the state counter exceeds a pre- 
determined threshold value Th (S44), attribute informa- 
tion added to an area is rendered "fusion "-possible 
(S45). All of the areas are checked, and thereafter, fusion 
processing is performed (S46). 

Area fusion processing in the step S46 will be 
explained with reference to a flow-chart shown in FIG. 
1 9. At first, all the areas are subjected to searching (S51), 
attribute information of areas is read (S52), and whether 
or not fusion is possible is checked (S53). If fusion is pos- 
sible, attribute information of close areas is checked as 
to whether or not fusion is possible (S53). If fusion is pos- 
sible, parameters of an area being processed are pro- 
vided for an area to be fused (S55). With respect to areas 
on which fusion has already been performed, attribute 
information is returned to a normal state, and this oper- 
ation is performed on ail the areas (S56 and S57). 

In fusion of areas, fusion is performed as shown in 
FIG. 16. for example. In this case, fusion of areas is car- 
ried out such that a new area becomes rectangular 
again. Specifically, three close areas, i.e., an area adja- 
cent to the target area in the right side thereof, an area 
below the target area, and an area below the adjacent 
area in the right side of the target area are checked. 

In the above, area division and fusion processing 
have been explained. However, if changes occur at a low 
frequency with in an area, with respect to each of areas 
in an image, it is possible to reduce the entire processing 
amount by reducing the number of times for which 
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processing is performed on related areas, without per- 
forming fusion processing as described above. 

In addition, if the number of areas changes by 
repeating division and fusion of areas, replacement of 
numbers of areas is performed if necessary. Replace- s 
ment of the numbers of areas is performed, for example, 
in a method of checking the coordinate value of the left 
upper corner which each area has as a parameter in the 
order from the left upper area to the right lower area in 
an image, thereby assigning numbers thereto. 10 

Although the above explanation has been made to 
a case in which fusion processing is performed subse- 
quently after division processing of areas is performed, 
the processing may be performed in a reverse order, or 
the division processing of areas and the fusion process- is 
ing areas may be performed in parallel. 

In the following, a third embodiment will be explained 
with reference to the drawings. The third embodiment is 
an example in which an abnormal state detect apparatus 
according to the present invention is applied to a paper 20 
material dealing apparatus, and in which a self -diagnosis 
function is supplied by a mark. 

FIG. 20 schematically shows the structure of a paper 
material dealing apparatus according to the present 
invention. Specifically, a supply section 1 supplies paper 25 
materials P such as stocks, mails, or the like, one after 
another. Paper materials P thus supplied are conveyed 
by a convey path 2, and distributed into branch convey 
paths 4 and 5 through a distribution gate 3 provided at a 
rear end portion of the convey path 2. Compilation sec- 30 
tions 6 and 7 are respectively provided at rear end por- 
tions of the branch convey paths 4 and 5, so that 
conveyed paper materials P are compiled and stored. 

Note that a determination section 8 for determining 
the kinds of paper materials P by optically reading image 35 
information on the paper materials P is provided in the 
middle of the convey path 2, and the distribution gate 3 
is switched and controlled in accordance with determi- 
nation results of the determination section 8. 

Meanwhile, an ITV camera 10 is provided at each of 40 
specific positions of the convey paths 2, 4, and 5 (i.e., 
within ranges in which an abnormal state is estimated to 
occur). For example, a predetermined range of a position 
of a distribution gate 3 is set as a monitor area 9, and an 
ITV camera 10 an image pick-up means for picking up 45 
an image within the monitor area 9 is provided. 

An image signal picked up by the ITV camera 10 is 
sent to a processing device 12 through a transmission 
path 1 1 , and processing for detecting abnormal convey 
of a paper material P is performed. When abnormal con- so 
vey is detected, an alarm device not shown is operated, 
or a figure or a letter indicating an abnormal state is dis- 
played on a display device 13. 

FIG. 21 is a block diagram of an example of a struc- 
ture according to the third embodiment, and mainly ss 
shows the processing device 12. This embodiment 
shows a case in which a self-diagnosis function is pro- 
vided for an image pick-up system. The basic structure 
of this embodiment is the same as that of FIG. 1, and 



therefore, only those portions of the structure which are 
different from the structure of FIG. 1 will be explained 
below. 

A detect processing section 25 mainly comprises, 
for example, a CPU and the like. This processing section 
25 performs differentiate calculations on a plurality of 
images (e.g., two images) taken in by an image memory 
23 on the time base, and extracts a change area between 
consecutive images on the time-base which are stored 
in the image memory 23, thereby to detect abnormal 
convey of paper materials P. Further, this section 25 
sends the results of detection to a display device 13 to 
display the results, and drives an alarm device 26 if nec- 
essary. 

Otherwise, this detect processing may use image 
division processing according to the above first and 
embodiments. 

Further, the section 25 is arranged so as to send 
such information as monitor section monitor signals 27 
to a host control device 28, thereby to control the entire 
apparatus. 

The detect processing section 25 is connected 
through a timing generate section 29 with a mark gener- 
ate section 30 as a specific mark generating means. The 
mark generate section 30 generates a mark in a monitor 
area 9 when self-diagnosis for an image pick-up system 
is performed. 

As a specific method of generating a mark in the 
mark generate section 30, for example, there is a method 
of using a mark M which can be electrically turned on/off 
like an LED (light emitting diode) or a method of making 
a mark M appear by mechanically opening/closing a 
shutter, as shown in FIG. 23. In this case, a change real- 
ized upon a mark generating instruction may be either 
ON/OFF (or "invisible" to "visible") or OFF/ON (or "visi- 
ble" to "invisible"). 

As a method of generating a mark other than those 
described above, such a method which enables a 
change to locally occur can be used. 

In addition, in case where a shutter is used as a 
mark, the shape which appears when the shutter returns 
back is explained as a circle in this example, but this 
shape is not limited to a circle in any cases. 

Next, the processing for performing self-diagnosis 
as to whether or not an image pick-up system is regularly 
operating will be explained with reference to flow-charts 
shown in FIG. 24. Image input operation from an ITV 
camera 10 is started (S101), and a self-diagnosis time 
measurement counter is reset (S102). The detect 
processing section 25 performs time-based differentiate 
processing on consecutive images on the time-base; 
which are taken in by the image memory 23. A difference 
between two consecutive images in the image memory 
23 is obtained (S103). and noise processing (S105) is 
performed on a result obtained by performing binarize 
processing (Si 04) on the result obtained above as the 
difference. Finally, whether or not concentrated change 
areas exist in an image is checked (S106). If such 
change areas exist, there is a possibility that the appa- 



7 



13 



EP 0 703 710 A1 



14 



ratus is moving within a monitor area 9. and therefore, 
the flow returns to the differentiate processing in the step 
S103. The operation as described above is repeated. 

With respect to noise reduction, it is possible to 
achieve noise reduction, for example, by eliminating iso- 5 
lated points from a binary image by means of compres- 
sion and expansion processing. 

If an abnormal state is not detected in the step S106, 
the detect processing section 25 makes a mark as 
described above generated in the monitor area 9 by 10 
sending a mark generate instruction to a mark generate 
section 30 through a timing generate section 29. 

Then, the detect processing section 25 takes in an 
image in the monitor area 9 again (S108), and performs 
detect processing by a method of time-based difference is 
using a plurality of images taken in by the image memory 
23, like detection of convey abnormal state as described 
above, or by a method of image division detect process- 
ing as specifically explained above (S109), thereby to 
check whether or not a change due to a mark is detected 20 

(51 10) after generation of a mark is instructed. If a mark 
is detected, the image pick-up system is determined as 
normally operating, and the state that the system is nor- 
mally operating is displayed on the display device 13 

(51 11) . Then, the mark generate instruction is released 25 
(S115). 

If a mark is not detected in the step S1 10, the con- 
tents of the self diagnosis time measurement counter is 
updated by VP (S112), and thereafter, whether or not 
the contents of the counter is larger than a predeter- 30 
mined value previously set is determined (S1 13). If the 
contents is smaller than the predetermined value, the 
flow returns to the image take-in processing in the step 
S108, and the same operation as described above is 
repeated. 35 

In the step Si 13, if the contents of the counter is 
larger than the predetermined value, it is determined that 
there is a defective in an input from the image pick-up 
system (S1 14), and the state that a defective exists in an 
input is displayed on the display device 13. Then, theflow 40 
goes to the step S1 15 and releases the mark generate 
instruction. 

Specifically, if generation of a mark cannot be 
detected when a predetermined time has passed since 
a instruction of generating a mark was supplied, it is 45 
determined that there is a defective in an input from the 
image pick-up system and a notification is supplied. If a 
mark is detected within the predetermined time, it is 
determined that the image pick-up system is normally 
operating and a notification is supplied. The successive 50 
processing is then continued. 

FIG. 25 is a block diagram of an example of a struc- 
ture mainly showing a processing device 12 according 
to a fourth embodiment of the present invention. This 
embodiment explains a case of comprising a self-diag- 55 
nosis function for a detect function of the image pick-up 
system, and differs from the third embodiment in that a 
false abnormal state generate section 31 is provided in 



place of a mark generate section 30 in the third embod- 
iment of FIG. 21. 

Specifically, the false abnormal state generate sec- 
tion 31 generates a false abnormal state within a monitor 
area 9 when self -diagnosis for the detect function is per- 
formed. As a specific generate method of generating a 
false abnormal state in the false abnormal state generate 
section 3 1 , for example, there is a method in which a false 
object 43 similar to a paper material P to be actually con- 
veyed is installed at an end portion of an arm 42 rotated 
around a shaft 41 as a fulcrum, as shown in FIG. 26, and 
the arm 42 is rotated so as to insert the false object 43 
into the monitor area 9 when a generation instruction is 
supplied, thus generating a false abnormal state. 

In the above example, explanation has been made 
to a method in which a false object 43 is installed on an 
end portion of the arm 42. However, it is possible to use 
another method than the method described above, as 
long as such a method can generate a false abnormal 
state. 

In the next, explanation will be made to processing 
for performing self-diagnosis as to whether or not the 
detect function is normally operating (i.e., whether or not 
an abnormal state occurring in a monitor area 9 can be 
correctly detected in this case), with reference to flow- 
charts shown in FIG. 27. Image input operation from the 
ITV camera 10 is started (S121), and the self-diagnosis 
time measurement counter is reset (S122). 

Then, the detect processing section 25 performs 
time-based differentiate processing on consecutive 
images on the time-base, which are taken in by the image 
memory 23. A difference between two consecutive 
images in the image memory 23 is obtained (S123), and 
noise processing (S125) is performed on a result 
obtained by performing binarize processing (S124) on 
the result obtained above as the difference. Finally, 
whether or not concentrated change areas exist in an 
image is checked (S126). If such change areas exist, 
there is a possibility that the apparatus is moving within 
a monitor area 9, and therefore, the flow returns to the 
differentiate processing in the step S103. The operation 
as described above is repeated. 

With respect to noise reduction, it is possible to 
achieve noise reduction, for example, by eliminating iso- 
lated points from a binary image by means of compres- 
sion and expansion processing. 

If an abnormal state exists in the monitor area 9 

(5126) , the flow returns to image take-in processing in 
the step S123, and the same processing as above is 
repeated. 

If an abnormal state is not detected in the step 126, 
the detect processing section sends a false abnormal 
state generate instruction to a false abnormal state gen- 
erate section 31 through a timing generate section 29 

(5127) , thereby to make a false abnormal state gener- 
ated in the monitor area 9, as described above. 

Then, the detect processing section 25 takes in an 
image in the monitor area 9 again (S128), and performs 
detect processing by a method of time-based difference 
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using a plurality of images taken in by the image memory 
23, like detection of convey abnormal state as described 
above, or by a method of image division detect process- 
ing as specifically explained above (S129), thereby to 
check whether or not an abnormal state is detected 
(S130) after generation of an abnormal state is 
instructed. If an abnormal state is detected, the detect 
function is determined as normally operating, and the 
state that this function is normally operating is displayed 
on the display device 13 (S131). Then, the abnormal 
state generate instruction is released (S132). 

If an abnormal state is not detected in the step S130, 
the contents of the self diagnosis time measurement 
counter is updated by (S133), and thereafter, 
whether or not the contents of the counter is larger than 
a predetermined value previously set is determined 
(S1 34). If the contents is smaller than the predetermined 
value, the flow returns to the image take-in processing 
in the step S128, and the same operation as described 
above is repeated. 

In the step S134, if the contents of the counter is 
larger than the predetermined value, it is determined that 
there is a defective in the detect function or that there is 
a defective in an input from the image pick-up system, 
and the state that there is a defective is displayed on the 
display device 13 (S135). Then, the flow goes to the step 
S132 and releases the abnormal state generate instruc- 
tion. 

Specifically, if an abnormal state cannot be detected 
when a predetermined time has passed since a instruc- 
tion of generating an abnormal state was supplied, it is 
determined that there is a defective in the detect function 
or that there is a defective in an input from the image 
pick-up system and a notification is supplied. If an abnor- 
mal state is detected within the predetermined time, it is 
determined that the detect function is normally operating 
and a notification is supplied. The successive processing 
is then continued. 

In the next, a fifth embodiment of the present inven- 
tion will be explained below. This embodiment explains 
a case of comprising an automatic setting function of a 
monitor area 9. 

FIG. 28 shows four marks Ml, M2, M3, and M4 
placed in a monitor area 9 to be monitored according to 
the third embodiment shown in FIG. 20. This embodi- 
ment has an object of securely monitoring the area sur- 
rounded by the marks M1, M2, M3, and M4. In the 
following, the area surrounded by these marks will be 
referred to as a monitor area 9, and explanation will be 
made supposing that four marks are set. 

When four marks M1 , M2, M3, and M4 placed in the 
monitor area 9 are picked up and taken in as an image 
by an ITV camera 10, these marks are viewed in several 
different ways due to dislocations of the ITV camera 10 
or the likes. For example, FIG. 29A shows a norma! state 
in which the monitor area 9 is positioned substantially in 
the center of the camera view field. FIG. 29B shows a 
state in which the monitor area 9 is positioned within the 
camera view field, with being shifted from the center of 
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the view field. FIG. 29C shows an example in which the 
monitor area 9 is out of the camera view field, and in this 
case, it is difficult to carry out monitoring which will 
achieve the object of this embodiment. 

5 In the following, the setting operation for setting a 

monitor area 9 will be explained with reference to flow- 
charts shown in FIGS. 30A and 30B. Image input oper- 
ation from an ITV camera 10 is started (S141), and 
image data is taken in by the image memory 23 (S143). 

10 In this state, the number (i.e., "4 H in this case) of marks 
used for setting a monitor area 9 is previously registered 
(S142). 

Then, the detect processing section 25 performs 
processing for detecting marks by a method of time- 
rs based difference using a plurality of images taken in by 
the image memory 23, like detection of convey abnormal 
state as described above, or by a method of image divi- 
sion detect processing as specifically explained above 
(S144), thereby to calculate the number m of marks thus 

20 detected (S1 45). Then, the calculated number m is com- 
pared with the number n of marks previously set (S146, 
S147, and S148). If the number of detected marks is 3 
or more, as a result of this comparison, it is possible to 
set a monitor area on the basis of these marks, and 

25 therefore, the area to be monitored is set on the basis of 
the positions of these marks. 

Specifically, if the number of detected marks is 4, the 
detect positions are recorded (Si 49), and thereafter, the 
monitor area 9 is set on the basis of the respective posi- 

30 tions of the four marks (S 150). If the number of detected 
marks is 3, detected positions thereof are recorded, and 
thereafter, the position of a fourth mark is estimated on 
the basis of the respective positions of the three marks 
(S152). If the position of the fourth mark is estimated, the 

35 processing goes to the step S150 and sets a monitor 
area 9 is set 

Otherwise, if the number of detected marks is 2 or 
less (although at least three or more marks are neces- 
sary), it is not possible to set an area except for several 

40 specific cases. In this case, it is therefore considered that 
the image pick-up system cannot grasp marks, and an 
instruction for correcting the setting of the image pick-up 
system is supplied. 

Specifically, if the number of detected marks is 2, 

45 whether or not the two marks are corners on the diagonal 
line of a monitor area is determined (S153). If yes, the 
detected positions thereof are recorded (S154), and 
thereafter, positions two other remaining marks are esti- 
mated on the basis of the respective positions of the two 

so marks (S155). If those positions are estimated, the 
processing goes to the step S150 and a monitor area 9 
is set. 

In addition, if the number of detected marks is 2 or 
less, it is determined that the image pick-up system is 
55 abnormal (S156), and a notification indicating that set- 
ting of a monitor area is impossible is displayed on the 
display device 13 (S157). Further, if detected two marks 
are not two corners on a diagonal line, in the step S1 53, 
the processing goes to the step S157, and a message 
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indicating that setting of a monitor area is impossible is 
displayed. 

Although the above explanation has been made to 
a case where a monitor area is set by supposing that the 
number of marks is four, the number of marks is not espe- s 
cially limited as long as the number of marks is 2 or more. 
In addition, it is possible to use the mark explained in the 
third embodiment described above, as an example of 
marks. 

As has been explained above, according to the first w 
embodiment of the present invention, processing with 
use of information concerning the entire of an image is 
not performed, but an image is divided into a plurality of 
areas, and processing using characteristic amounts in 
respective areas is performed. Besides, analysis is per- is 
formed on the time-based changes in the characteristic 
amounts. By thus roughly dividing an image intentionally, 
it is possible to absorb small changes such as vibration < 
and the like, and to concentrate to extraction of only large 
changes (such as a movement of an object in an image), 20 
so that changes in an image, occurrence of an abnormal 
state, and the likes can be extracted without making com- 
parison with templates. 

In addition, it is possible to absorb influences from £ 
vibration of an ITV camera or the like, in the step of divid- 25 
ing an image into a plurality of areas, so that processing 
such as noise reduction is not required. 

According to the second embodiment, it is possible 
to perform fine detection of an abnormal state and abnor- 
mal state detect processing at a high processing speed. 30 
by further dividing the divided areas or by fusing the 6 
divided areas with each other. 

In addition, according to the third and fourth embod- 
iments as described above, it is possible to absorb a 
large or small shift of the installation position of an ITV 35 
camera and to notify a system manager of the portion 
where a defective occurs on the basis of a diagnosis 7 
result, by comprising a self-diagnosis function of con- 
firming that an image pick-up system or a detection func- 
tion is normally operating. Therefore, labor for 40 
inspections and adjustment services can be greatly 
reduced. 

According to the fifth embodiment, the monitor area 8. 
as a target can be automatically corrected, and there- 
fore, erroneous detection is not easily caused even when 45 
the orientation of an ITV camera is more or less changed. 

Claims 



1. 



An image data processing apparatus characterized 
by comprising: 

means (10, 21 , S2) for continuously receiving 
a image of a first area (FIG. 3, (a)) to be processed, 
and for converting the image into a first image data 
item; 

means (25, 23, S3) for dividing the first image 
data into a plurality of second image data items cor- 
responding to a plurality of second areas (FIG. 3, 
(b)) which are smaller than the first area; and 
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means (23, 24, 25, S4) for performing prede- 
termined processing for each of the second image 
data items so as to determine an abnormal state in 
the first area. 

. An apparatus according to claim 1 , 

characterized in that the detecting means includes 
means (23, 24, 25, and S4) for detecting a time- 
based change in a characteristic amount of the 
image for each of the second image data items. 

. An apparatus according to claim 1 , 

characterized in that the detecting means includes 
means (23, 24, 25, and S4) for detecting a time- 
based change in an average density amount of the 
image for each of the second image data items. 

An apparatus according to claim 1 , 
characterized in that the detecting means includes 
means (23, 24, 25, and S4) for detecting a time- 
based change in a density dispersion value of the 
image for each of the second image data items. 

An apparatus according to claim 1 , 
characterized in that the detecting means includes 
means (23, 24, 25, and S4) for detecting a time- 
based change in randomness obtained as a result 
of differential binarization of the image for each of 
the second image data items. 

An apparatus according to claim 1 , 
characterized in that the detecting means includes 
means (23, 24, 25, and S4) for detecting a time- 
based change in a hue amount of a color image for 
each of the second image data items. 

An apparatus according to claim 1 , 
characterized in that the detecting means includes 
means (23, 24, 25, and S4) for detecting a time- 
based change in chroma of a color image for each 
of the second image data items. 

An apparatus according to claim 1 , 
characterized in that the detecting means includes 
means (23, 24, 25, and S4) for detecting a time- 
based change in brightness of a color image for each 
of the second image data items. 

An apparatus according to claim 1 , 
characterized in that the detecting means includes 
means (23, 24, 25, and S4) for detecting a time- 
based change amount in a characteristic amount of 
the image for each of the second image data items, 
for determining abnormality of the second image 
data items on the basis of the change amount; and 
means for determining that the first area is in an 
abnormal state if the number of such adjacent items 
among the second data items which are determined 
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as being abnormal is equal to or more than a prede- 
termined amount. 

10. An image data processing apparatus according to 
claim 1 , characterized in that the performing means s 
includes; 

first detecting means (23, 24, 25, S4) for 
detecting a time-based change amount in a charac- 
teristic amount for each of the second image data 
items; 10 

first finding means (S31 to S34) for finding 
unstable image data items from the second image 
data items, the unstable image data items have the 
time-based change amount in the characteristics 
remaining to or less than a first predetermined value, is 
and have an occurrence frequency of changes 
exceeding a predetermined number of times for a 
first predetermined period or more; 

second dividing means (S31 to S34) for divid- 
ing the first image data item so as to form third image 20 
data items by dividing the unstable image data items 
when the first finding means finds the unstable 
image data items; 

second detecting means (S4) for detecting a 
time-based change amount in a characteristic 25 
amount for the third image data items; 

second finding means (S31 to S34) for finding 
plain image data item from the second image data 
items, the plain image data items are adjacent to 
each other and which have the time-based change 30 
amount in the characteristics remaining equal to or 
less than a second predetermined value for a sec- 
ond predetermined period or more; 

third dividing means (S31 to S34) for dividing 
the first image data item so as to form fourth image 35 
data items by fusing the plain image data items when 
the second finding means finds the plain image data 
items; 

third detect means (S4) for detecting a time- 
based change amount in a characteristic amount for 40 
the third image data items; and 

means for determining an abnormal state of 
the first area on the basis of the change amounts 
detected by the first, second, and third detecting 
means. 45 

11. An apparatus according to claim 10, 
characterized by further comprising: 

means (29, 30, and S107) for generating a 
recognizable mark in the first area at a predeter- so 
mined timing; and 

means (29, 30, and S 1 08 to S 1 1 1 ) for detect- 
ing the mark generated by the generating means so 
as to determine that an inverting function of the 
image data processing apparatus regularly oper- 55 
ates. 

12. An apparatus according to claim 10, 
characterized by further comprising: 
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means (29, 30, and S107) for generating an 
abnormal state in the first area at a predetermined 
timing; and 

means (29, 30, and S110 and S111) for 
detecting the abnormal state generated by the gen- 
erating means so as to determine that an abnormal 
detection function of the image data processing 
apparatus regularly operates. 

13. An apparatus according to claim 10, 
characterized by further comprising: 

means (29, 30, and S107) for generating a 
plurality of marks indicating a range of an area to be 
monitored, in the first area at a predetermined tim- 
ing; and 

means (23, 24, 25, S110, and Si 11) for 
detecting the marks generated by the generating 
means so as to determine that an inverting function 
of the image data processing apparatus regularly 
operates. 

14. An image data processing method characterized by 
comprising the steps of: 

continuously receiving a image of a first area 
(FIG. 3, (a)) to be processed, and converting the 
image into a first image data item (S2); 

dividing the first image data item into a plu- 
rality of second image data items corresponding to 
a plurality of second areas (FIG. 3, (b)) which are 
smaller than the first area; and 

determining an abnormal state in the first 
area, by performing predetermined processing for 
each of the second image data items. 

15. An image data processing method according to 
claim 14, characterized in that the determining step 
includes a step of detecting a time-based change 
amount in a characteristic amount of the image for 
each of the second image data items, and of deter- 
mining an abnormal state in the first area on the 
basis of the change amount 

16. An image data processing method according to 
claim 14, characterized by further comprising the 
steps of; 

finding (S31 to S34) unstable image data 
items from the second image data items, the unsta- 
ble image data items have the time-based change 
amount in the characteristics remaining to or less 
than a predetermined value, and have an occur- 
rence frequency of changes exceeding a predeter- 
mined number of times for a predetermined period 
or more; 

a second dividing step (S31 to S34) of divid- 
ing the first image data item so as to form third image 
data items by dividing the unstable image data items 
when the finding step finds the unstable image data 
items; 

a detecting step (S29) of detecting a time- 
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